When the Java Platform was original designed it was conceived to be a secure operating environment for untrusted applications, allowing a user to operate applications from untrusted third parties in a quarantined sandbox environment. However, over the years the Java Platform has gained a contrary reputation, as one of the most insecure and vulnerable software environments in the world.
The Java Platform today is very different from the time of its inception. The Java Platform is today in its 9th major release (Java SE 8), and can comprise up to 5 million lines of source-code. The volume of source-code in the Java Platform—far more than other similar platforms—has at least in part contributed to the poor security profile and frequent vulnerability discoveries of recent years. Such a large source-code-base is beyond the ability of any one programmer to reason about secure and insecure operations, and this has in turn led to new vulnerabilities and exploitable weaknesses which arise from the unintended consequences of “code-bloat”.
At the root of Java's security problems is the security model designed into the Java Platform—called the “Java Security Architecture”. The original security model provided by the Java Platform is known as the sandbox model, which existed in order to provide a very restricted environment in which to run untrusted code obtained from the open network. The essence of the sandbox model is that local code is trusted to have full access to vital system resources (such as the file and memory systems) while downloaded remote code (an applet) is not trusted and can access only the limited resources provided inside the sandbox. This sandbox model is schematically illustrated in FIG. 1.
As seen in FIG. 1, the JAVA API class library includes for example an “access function” F which is for accessing valuable resources of the host computer. The access function is responsible for determining whether access to a valuable resource is guarded by some security policy, and if so, to consult the java.lang.SecurityManager class to vet such calls and either allow or disallow them. However, in practice, it is fairly easy to manipulate the access function F to bypass the SecurityManager checks and proceed without SecurityManager policy in effect. The effect of such an attack is that the security manager is effectively bypassed or nullified, and so the SecurityManager fails to reliably undertake the security control role it is expected to perform.
In this prior art scheme, the design flaw from which the poor security controls arise, is that the SecurityManager class, which is where the security control policy is implemented and applied, is only used in an “advising” capacity by access function F. In otherwords, access function F queries the SecurityManager class whether access function F is permitted to operate, and the SecurityManager class returns an indication of whether access is allowed or not. Thus, the SecurityManager serves a purely advisory role to access function F, and is not capable of directly controlling the operation of access function F, and therefore is not able to directly control operations performed upon valuable resources. As a result, a bug or discrepancy in the code or behaviour of access function F—or contextual manipulation by an Application class—can cause access function F to omit to consult the SecurityManager class, and so proceed to allow unsafe access to a valuable resource.
In this security model, an Application Class such as a “ClassLoader” class defines a local name space, which can be used to ensure that an untrusted applet cannot interfere with the running of other programs. Access to crucial system resources was intended to be mediated by a “SecurityManager” class, in combination with “ClassLoader” classes, to restrict the actions of a piece of untrusted code to the bare minimum.
What was significant in this software design, was that the security-enforcement capability—the “SecurityManager” class and one or more “ClassLoader” classes—were implemented side-by-side with the application code these classes were intending to mediate. In other words, these classes ran inside the same sandbox as the untrusted application classes, and used the same Java Platform Application Programming Interfaces (API's) as the untrusted classes.
The consequence of this security design, has been the emergence of a large number of discovered vulnerabilities where application classes, operating side-by-side with the security-enforcing SecurityManager class, can bypass the security control facilities of the SecurityManager class to do any of: (i) uninstall the SecurityManager, (ii) replace the SecurityManager, (iii) undertake operations or actions that the SecurityManager believes are from itself or it's associated ‘trusted’ code, or (iv) undertake operations or actions that the SecurityManager is unaware are taking place
Thus, the Java Security Architecture—relying on the SecurityManager class facility—is not able to reliably ensure that an application, and/or the Java API class library on which the application operates, do not undertake undesirable actions that an administrator might want to control or restrict, such as for example denying the application and Java API class library from reading a file of a specified directory. Due to the design flaws of the Java Security Architecture, it is possible to bypass or circumvent the SecurityManager controls and thereby break out of the security controls sought to be enforced by the application administrator.
For several examples of vulnerabilities and weaknesses in one or more versions of the Java API classlibrary where the SecurityManager control checks can be bypassed, circumvented, skipped, or removed altogether see http://www.security-explorations.com/materials/se-2012-01-report.pdf.
This “sandbox” security model is very different from the security model used—and long established—in operating systems. Almost all commercial operating systems rely on “protection rings” to enforce security. FIG. 2 illustrates a typical prior art security ring arrangement.
In this security model, a “ring” is a layer of privilege within the software architecture. Rings R0, R1, R2, . . . etc. are arranged in a hierarchy from most privileged (most trusted, usually numbered zero), to least privileged (least trusted, usually with the highest ring number). On most operating systems “ring 0” is the level with the most privileges and interacts most directly with the physical hardware such as the CPU and memory.
Importantly, a strict and formally defined interface (sometimes called “a gate”) such as G0, and G1 between rings is provided to allow an outer-ring to access an inner ring's resources in a predefined manner, as opposed to allowing arbitrary usage. Thus, an outer (less privileged) ring can only access valuable resources by requesting an inner ring to perform the sought action on the outer ring's behalf. This means that inner rings act as mediating proxies for value resources on behalf of outer rings, and that outer rings do not have direct access to valuable resources or the operations that can be applied directly to outer resources. Correctly gating access between rings improves security by preventing programs from one ring or privilege level from misusing resources intended for programs in another.
More recently, with the emergence of x86 hypervisors like VMware and Xen, there has been a renewed interest in this “protection ring” structure. In a hypervisor platform, different operating system “guests” operate inside less-privileged rings, which can only interact with the physical computer's resources through the defined interface (“gate”) that the hypervisor exports. The hypervisor thus serves as a proxy for the valuable physical computer resources, and therefore can reliably mediate safe and secure access to these valuable resources which cannot be accessed directly by untrusted “guest” software.
Extending this concept further, hypervisor developers used the protection rings and the formal “gated” interface that they defined, to implement emulation facilities for functions which were either too insecure or untrusted to occur on the real physical server. Examples of this are interacting with network interfaces and storage controllers.
A frequently used term to describe the most privileged ring is “supervisor mode”.
Unfortunately, the way the Java Platform and its components (specifically the Java API class libraries, and the Java Virtual Machine, or JVM) are implemented mean that Java does not operate with privilege rings, and so does not operate with mediating proxies that can control and enforce the safe use of valuable resources. Instead, Java application classes, as well as any of the Java API class libraries, can directly call privileged “native” methods inside the JVM or OS libraries that access valuable resources directly and in an uncontrolled manner, and the only opportunity to enforce security controls is in the advisory check of the SecurityManager class by the Java API class libraries. As a result, it has been considered impossible to introduce “protection rings” into the Java Platform without breaking binary compatibility with existing Java applications that have been written.